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ABSTRACT 


A  "pooling  sets"  type  of  algorithm  is  developed  and  shown  to 
be  valid  for  computing  an  isotonic  regression  function  for  a  gen¬ 
eral  quasi-order.  The  method  is  direct  and  intuitive.  The  algo¬ 
rithm  works  best  when  the  quasi-order  is  complex  and  the  objective 
function  is  nearly  isotonic.  An  example  is  worked  out  in  detail. 
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Introduction . 


Consider  X  to  be  a  finite  set  which  we  denote  as 

X  =  {x, ,  x-,...,x  }.  Let  <  be  a  binary  relation  on  X  which  is  a 
l  i  m 

quasi-order,  i.e.  <  is  reflexive  and  transitive,  (see  Barlow, 
Bartholomew,  Bremner,.  and  Brunk  (1972)  p.24).  We  say  a  real¬ 
valued  function  f(-)  defined  on  X  is  isotonic  with  respect  to 
"<"  if  x  <  y  implies  f(x)  <_  f  (y)  .  A  problem  that  arises  in  many 
different  contexts  is  the  following:  For  a  given  positive  function 
w(x) ,  called  the  weight  function,  and  an  arbitrary  function  g(x) , 
find  that  function  g*(x)  on  X  which  is  isotonic  with  respect  to  a 
particular  quasi-order  "< "  and  minimizes 

m 

Z  (g (x. )  -  h(x. ) ) Mx. ) 
i“l  1  11 

over  all  isotonic  functions  h(x) .  This  is  referred  to  as  the  iso¬ 
tonic  regression  problem.  Much  of  the  importance  of  this  problem 
hinges  on  the  fact  that  is  solves  many  seemingly  very  different 
optimization  problems  (see  Barlow  et.al.  (1972)  and  Barlow  and 
Brunk  (1972) )  . 

This  is  of  course  a  quadratic  programming  problem  with  spe¬ 
cial  types  of  linear  constraints.  In  certain  cases,  the  solution 
is  well  known  and  easily  computed. 

i)  Simple  ordering  case.  When  "<"  is  also  a  simple  ordering 
(reflexive,  transitive,  an ti -symmetric ,  and  each  pair  of  elements 
is  comparable) ,  the  problem  is  greatly  simplified.  One  of  the 
more  common  situations  is  when  X  consists  of  real  numbers  and  "<" 
is  the  usual  ordering  of  the  real  numbers.  In  this  case,  a  func¬ 
tion  is  isotonic  iff  it  is  nondecreasing.  In  this  situation. 


g*(x)  can  be  effected  in  the  following  way:  Reduce  the  problem  to 
fewer  dimensions  by  pooling  adjacent  points  where  reversals  occur. 
That  is,  choose  x.  ,x.  such  that  x.  <  x.,  g(x.)  >  g(x.),  and  there 
does  not  exist  y  such  that  x^  <  y  <  x^  (if  no  such  pair  exists, 
then  g(x)  *  g*(x)).  Now  consider  the  same  problem  where  x^  and 
Xj  are  considered  as  one  point  with  weight  w(x^)  +  w(x^) ,  func¬ 
tional  value 

gUx^,  Xj})  *  [g  (x^)  w(x^)  +  g  (x.. )  w  (x^  )  ]  [w  (x^  +  wfx.)!*1, 
and  the  natural  ordering  {x.,x.}  <  x.  if  x-  <  x.  and  x.<{x.,x.} 

X  j  *•  J  **  JC  K**  X  J 

if  x^  <  x^.  This  reduction  process  is  continued  until  a  function 
g  is  obtained  with  no  reversals  .  Then  g*(x)  *  g(x)  if  x  is  one 
of  the  points  making  up  the  pool  x.  This  is  the  "Pool  Adjacent 
Violators  Algorithm"  first  discussed  by  Ayer  et.al.  (1955) . 
ii)  Rooted  trees .  A  more  general  class  of  partial  orderings  that 
is  important  in  some  problems  involves  the  concept  of  rooted  trees. 

In  a  rooted  tree,  except  for  a  single  element,  the  root,  which  has 
no  predecessor,  each  element  has  exactly  one  immediate  predecessor. 

(We  say  x^  is  an  immediate  predecessor  of  x^  if  x^  <  x^  and  there 
does  not  exist  an  element  y  such  that  x^  <  y  <  x^)  . 

Thompson  (1962)  has  shown  that  the  previous  scheme  used  for 
the  simple  ordering  case  works  providing  the  poolings  are  taken 
in  the  correct  order.  To  be  more  precise,  if  there  exists  an 
immediate  predecessor  to  x^,  say  x^  such  that  g(x^)  >  g(x^),  call 
the  point  x^  a  violator  (of  x4)  and  call  x^  a  violatee  (of  x^) . 

A  violator  x^  is  a  minimum  violator  if  there  does  not  exist  a  violator 
x.  such  that  g(x,  )  <  g(x.).  Thompson  showed  that  if  one  pools  a 
minimum  violator  to  its  violatee  (there  exists  only  one  in  the 
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rooted  tree  case) ,  redefines  the  partial  order  in  the  natural  way, 
and  continues  as  in  the  case  of  simple  ordering,  that  one  must  even¬ 
tually  arrive  at  g*(x) .  This  is  often  called  the  "Minimal  Violator 
Algorithm" . 

iii)  Partial  ordering  case.  For  general  partial  orders,  the  so¬ 
lutions  become  less  tractable.  A  closed  form  expression  for  g* 
described  in  Brunk  (1955)  involves  the  use  of  upper  and  lower  sets. 

'a)  A  c  x  is  an  upper  set  if  x^sA  and  x^  <  x^  implies  x^sA. 
b)  A  c  X  is  a  lower  set  if  x^eA  and  x^  <  x^  implies  x^eA. 

If  we  let  U  =  {A;  A  is  an  upper  set},  L  =  {A;  A  is  a  lower  set}, 
and  for  any  A  <=  X 

Z  g(t)w(t) 

.  ...  tsA 

Av  (A)  *  - , 

Z  w(t) 

'  t£A 


then 


g*Cx)  *  max  min  Av(A  n  B)  .  (1.1) 

AeU  Bel 

xeAnB 

Based  on  this  expression  for  g*(x),  there  exists  a  simple  algorithm 
(Minimum  Lower  Sets  Algorithm)  stated  in  Brunk,  Ewing,  Utz  (1957) 
for  computing  g* .  To  use  this  algorithm,  first  find  L^el  such  that 
Av(L^)  is  minimal.  Then  for  xeL^,  g*(x)  *  Av(L^) .  Next  find  L2el 
such  that  Av(L2  -  L^)  is  minimal.  Then  for  xeL2  -  L^,  g*  (x)  * 

Av(L2  -  L1) ,  etc.  This  is  repeated  until  *  X  and  g*  is  com¬ 
pletely  determined. 

This  algorithm  is  straightforward  and  direct,  but  does  have 
some  shortcomings. 

1)  The  class  of  ail  lower  sets  may  be  difficult  to  determine 
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and  enumerate  and/or  excessively  large.  For  example,  in  the 
natural  partial  ordering  for  the  grid  X  =  {  ( i ,  j )  ;  i*l,...,a, 
j  =  l,...,b}  defined  by 

(i»j)  <  (k,£)  iff  i  <_  k  and  j  <_  i , 

it  can  be  shown  by  combinatorial  techniques  that  there  are 
(a*k)  lower  sets.  This  number  becomes  very  large  as  a  and  b 

d 

become  large . 

2)  Computing  the  many  averages  Av(L)  for  all  lower  sets  may 
involve  a  large  amount  of  effort. 

3)  If  g  is  nearly  isotonic  (very  few  reversals) ,  as  much 
work  (in  fact  usually  more  because  the  L^,  Lj,  etc.  are  small 
and  hence  more  minimums  must  be  compared)  is  required  as  if 

g  is  far  from  isotonic. 

For  these  reasons,  a  scheme  similar  to  the  pooling  algorithm  of 
Thompson  for  the  rooted  tree  case  would  be  desirable.  Since  an 
arbitrary  quasi-ordering  allows  a  great  deal  more  diversity  than 
the  rooted  tree  situation,  it  is  not  surprising  that  the  general 
quasi-order  algorithm  is  more  complicated  than  for  the  rooted  tree 
case.  As  in  Thompson's  algorithm,  poolings  are  again  formed  with 
the  idea  of  reducing  the  dimension  of  the  problem.  However,  in 
this  case,  we  must  also  form  temporary  or  "working"  pools  which 
may  be  dissolved  later  in  the  algorithm. 

After  the  pooling,  our  new  problem  is  defined  in  the  natural 
way.  That  is,  if  the  points  x  and  y  are  pooled,  then  (x,y;  is 
treated  as  one  point  with  weight  w(x)  +  w(y)  and  functional  valve 
Av({x,y>) .  The  ordering  is  preserved  naturally;  i.e.  {x,y}  <  z  if 
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x  <  z  or  y  <  z  and  z  <  {x,v}  if  z<x  or  2<y. 

We  remark  that  poolings  are  to  be  made  only  between  violators 
and  their  immediate  predecessors. 

2 .  Justification  of  Algorithm. 

Since  the  algorithm  works  by  pooling  points  together,  we  need 
some  criterion  which  will  justify  using  the  pooling  mechanism. 

Lemma  2.1  is  basic  to  the  algorithm.  Lemma  2.1  is  similar  to  The¬ 
orem  2.5  of  Barlow  et.al.  (1972)  except  we  need  not  assume  that  x^ 
and  x2  are  "poolable". 

Lemma  2.1.  If  g*  denotes  the  solution  to  an  isotonic  regression 
problem  and  g  denotes  the  solution  to  the  isotonic  regression  prob¬ 
lem  where  x^  and  x2  are  pooled  to  form  y  (see  section  1  for  details 
of  the  pooling),  then  providing  g*(x^)  *  g*(x2)  ,  g  and  g*  agree 

A  A 

in  the  sense  that  g*(x)  *  g(x)  ,  x  ?  x^,x2  and  g*(x1)  -g*(x2)  *g(y). 

Proof.  Suppose  h  is  a  function  defined  on  X  such  that 
h(x^)  ■  h(x2)  *  c  and  h'  is  a  function  such  that  h' (x)  *  h(x)  , 
x  f  y  ,  and  h' (y)  *  c  .  Note  first  that  h  is  isotonic  on  X  iff 
h'  is  isotonic  for  the  pooled  problem.  Then  the  sum  of  terms  in 

m  2 

Z  (g (x. )  -  h(x. )  w(x. )  ( 2 

i_l  i  x  x 

which  involve  x^  and  x2  are 

2  2 
(g(x1)-c)  w(x^)  +  (g(x2)-c)  w(x2) 

=  (Av( {x1 ,x2 }) -c) 2 (w (x1 )  +  w ( x  2 ) )  +  c(x1 ,x2; 
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where  i>  does  not  depend  upon  c.  Since  the  value  of  c  that  mini¬ 
mizes  the  left  side  also  minimizes  the  right  side,  the  assumption 
that  g*(x^)  =  g^tx^)  implies  the  desired  result. 

Thus  if  we  have  points  x^  and  X£  such  that  x^  <  x^  and  x^  <  x^, 
we  know  that  we  may  pool  x^  and  and  consider  the  pooled  problem. 

An  algorithm  due  to  Von  Eeden  (see  Barlow  et.al  (1972) ,  page  90) 
is  of  interest  to  us.  In  this  scheme,  order  restrictions  are  num¬ 
bered  sequentially,  say  0^,  Let  g*  denote  the  solution 

to  the  smaller  problem  consisting  of  only  the  first  k  constraints . 

s  t 

Then  consider  the  k+1  constraint,  say  h(x.)  <  h(x.).  If 

i  --  2 

gkixi)  -  gk(xj}/  gk+l  =  gk ’  Otherwise,  g£+1(x.,)  =  9£+1(Xj>»  which 
effectively  reduces  the  problem  of  finding  g*+1  to  one  involving 
only  k  constraints.  By  induction,  one  can  obtain  g*  by  using  the 
above  result.  The  problem  lies  in  the  inefficiency  of  the  method, 
since  any  pooling  may  be  later  broken  up.  with  the  observation 
that  the  minimal  violator  always  stays  pooled,  we  may  greatly  im¬ 
prove  upon  this  scheme  for  the  isotone  regression  problem. 

If  A  c  X,  g(x) |A  will  denote  g  restricted  to  A.  We  can  also 
consider  our  isotonic  regression  problem  scaled  down  to  A.  That 


is,  for  x^,  x^sA,  x^  <  x^(in  A)  iff  x^  <  x^(in  X).  We  will  denote 
the  solution  of  our  restricted  problem  by  g*.  Of  course  g*|A  is 
the  overall  solution  restricted  to  A.  We  assume  that  <  is  a  par¬ 


tial  order  in  the  following  three  lemmas. 


Lemma  2.2.  If  L  is  a  lower  set,  then  g*  >_  <3*!L- 

Proof .  Note  first  that  the  upper  and  lower  sets  of  our  restricted 
problems  are  respectively 


Ltr  -  { An L ;  A  is  an  upper  set),  and 
Li 

=  {BnL;  B  is  a  lower  set}. 
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Thus ,  from  (1.1)  , 


gT*(x) 


=  max  min  Av ( A' -  3') 

a'£Ul  b^£Ll 

xeA'  n  B' 


=  max  min  Av(A  m  Bn  L ) 
AeU  Be  1 

xeA  i  3 


>  max  min  Av(A  n  B) 
AeU  Bel 

xeA  n  B 


=  g* (x) 


since  Bel  =s>  B  n  Lei  and  hence  L  contains  at  least  as  many  sets  as 

A  lemma  somewhat  similar  to  lemma  2.2  which  we  shall  need  is  the 
fallowing  one. 

Lemma  2.3.  If  A*  n  B*  c  C  =*  A1  n  Bx  where  A*  and  Ax  are  upper  sets, 
B*  and  B^  are  lower  sets,  and  i.or  all  xeA*  n  B* ,  g*(x)  *  Av(A*  n  B*)  , 
then  g*(x)  =  g*(x)  =  Av(A*  n  B*)  for  x  e A*  n  B*. 


Proof.  For  xeA*  n  B*  we  can  write 


g*(x)  =  max  min  Av(A  n  3  ”>  A.  n  B_  , 
AeU  Bel 

xeA  i  3 


>.  max  min  Av(A  i  A.  i  3) 
AeU  Be L  1 

xeA  ■>  3 


=  g* (x)  ,  since  A*  =  A*  ^  A.  • 
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Similarly , 


g*Cx)  =  min  max  Av(A  i  3  "i  A1  n  B,  ) 

3 eL  AeU  1 

xzA  '  3 

<.  min  max  Av(A  :  3  i  B.) 

3c  L  AeU  i 

xeA  r.  3 

=  g*(x) ,  since  3*  =  3*  n  3^. 

from  which  the  conclusion  follows. 

Note  that  this  lamina  implies  that  if  for  any  set  3, 

A  =  g*_1(3),  then  g*(x)|A  *  g*(x).  We  will  also  need  lemma  2.4. 

Lemma  2.4.  If  m^  is  a  minimal  violator  in  X,  g*(m^)  =  g*(v)  for 
some  immediate  predecessor  v  of  m^  (i.e.,  m^  can  be  pooled  with  one 
of  its  immediate  predecessors. 

Proof :  Assume  g*(m^)  >  g*(v)  for  all  immediate  predecessors  v  of 

m^ .  Thus  if  we  omit  all  restraints  of  the  form  h(Xj)  £  h(m^) ,  the 
reduced  problem  still  admits  the  solution  g* .  Let 

B  =  {x;  g(x)  <  g(m1)}  u  {m, }  . 

Since  g(x)  >  g  (m_  )  for  all  x  t  B  ,  we  must  have  gMx)  >_  g  (m.  )  (see 
1  3  1 

3runx  (1965)).  Thus  if  we  define 


g  (x  ) 


r  * 

.  ?  3(x)  , 
jg  (x)  , 


xs3 
xeB , 


g(x)  is  isotonic  for  the  reduced  problem.  To  see  this,  note  that 


for  the  original  problem,  there  can  be  no  violators  in  B  save 


since  is  a  minimal  violator,  and  hence,  for  the  reduced 

problem,  there  are  no  violators  in  3 .  No  point  in  B  can  be  a 

violatee,  since  the  corresponding  violator  would  have  to  be  in 

3  and  gf.  ^  g(m1)  >_  g(z)  ,  zeB.  Thus  g  satisfies  all  restrictions 
3  1 

involving  at  least  one  element  of  B  and,  from  its  definition, 
clearly  satisfies  all  restrictions  involving  elememts  both  in  B. 
However,  by  Van  Eeden's  algorithm  (see  first  part  of  this  sec¬ 
tion),  g  *  g* ,  which  is  a  contradiction,  since  a  violatee  v  of 
m^  belongs  to  B  and  thus  g^(v)  =  g*(v)  <  g*(m^)  =*  g(m^). 

Thus  the  minimal  violator  must  be  permanently  pooled  with 
at  least  one  of  its  immediate  predecessors.  The  following  the¬ 
orem  answers  the  question  of  "Which  one?"  and  serves  as  the  basis 
of  our  algorithm  for  computing  g*.  For  any  point  x,  we  define  the 
particular  lower  set  L(m^)  as  being  the  complement  of  {y;  x  <  y}  . 


Theorem  2.1.  if  m1  is  a  minimal  violator  and  v^  the  maximal 
immediate  predecessor  of  m.  with  respect  to  g*  .  (i.e. 


g£(mx)  (V  ±  9£(lBi)  (v)  for  311 

v  of  m1)  ,  then  gMn^)  =  g*(v). 

Proof .  Asstime  g*(m^)  >  g*(v^). 
some  immediate  predecessor  of  m. 


'  r 

other  immediate  predecessors 

From  lemma  2.4,  there  must  exist 
,  say  v,  such  that 


Noting  that 


g* (m1) 

{x;g* (x) =g* (v^) , 
g*(v1)  = 


*  g* (v)  .  (2.2) 

it  follows  from  Lemma  2.3,  that 
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Bv  definition, 


9't(m1)<vl>  i  9£(n.1)(v’  ' 
Finally,  Lemma  2.2  implies 


(2.4) 


5L(mx)(v)  -  gMv)  • 


(2.5) 


Combining  (2.3),  (2.4),  (2.5)  and  (2.2),  we  have 

9'1V1>  "  9L(m1)(''Xl  i  3£(m1>(v)  i  **  <v>  *  «*  lmX>  ' 
which  is  a  contradiction. 

3 .  The  Algorithm. 

Lemma  2 . 1  and  Theorem  2 . 1  suggest  a  pooling  algorithm  for 
obtaining  the  solution  to  an  isotane  regression  problem  with  a 
general  quasi-ordering.  That  is  first  reduce  the  quasi -ordering 
problem  to  a  partial  ordering  problem  by  sequentially  pooling 
points  x  and  y  where  x  <  y  and  y  <  x  and  redefining  the  ordering 
until  all  such  pairs  are  coalesced. 

If  one  can  solve  the  partial  ordering  problem  in  n 
dimensions  and  is  confronted  with  such  a  problem  in  n  +■  1  di¬ 
mensions,  he  has  only  to  locate  a  minimal  violator  ,  solve  the 
restricted  problem  over  L(m1)  and  pool  m^  with  the  resulting  maxi¬ 
mal  immediate  predecessor  (with  respect  to  g*(m  ))•  This  reduces 
the  problem  to  n  dimensions  which  can  be  handled  by  assumption. 
Actually  the  scheme  becomes  much  more  tractable  when  one  realizes 
that  one  does  not  have  to  begin  all  over  again  for  all  lower  points 


after  a  pooling  is  made.  Lemma  2.3  and  Lemma  3.1  are  particularly 
useful  in  this  regard. 


Lemma  3.1.  If  A  and  B  are  disjoint  and  if 


g  (x)  = 


g£(x)  , 

X  £  A 

g£(x)  , 

X  £  B 

then  9^uB(x)  =  g(x)  if  g  is  isotonic  over  the  set  A  u  B. 

Proof .  The  proof  is  obvious  by  writing  the  sum  to  be  minimized 

as  the  sum  over  A  plus  the  sum  over  B. 

We  note  from  Lemma  2.3  that  if  v  is  the  maximal  immediate 

predecessor  of  m  under  g*  ,  .  and  if 

L  (m) 

A  =  {yeL(m) :  g£(m) (y)  *  g£(n) <v) >  , 

then  g£(raj  (x)  *  g£(x)  for  all  xeA.  By  using  this  fact  with  lemma  3.1, 

it  is  often  easy  to  construct  g*  >.  for  the  next  minimal  violator  m' 

L  (m  ) 

Of  course  one  can  also  describe  the  algorithm  by  first  finding 
maximal  viola tees,  and  then  pooling  these  to  their  (restricted) 
minimal  violators.  For  large  sets,  a  combination  of  the  two  ap¬ 
proaches  is  sometimes  useful. 


4 .  An  Example. 

Consider  the  sixteen  points  in  the  plane 

x  =((i»  j);  i,j  =  1,  2,  3,  4}  with  the  usual  planar  ordering 
(i,  j)  <  (h,  k)  iff  i  <_  h  and  j  <_  k.  Assume  all  weights  are 
ecual  to  one  and  that 


i 
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Since  the  minimal  violator  is  (2,2)  and 


we  know  that  (2,2)  and  (2,1)  may  be  pooled  to  yield 


Now  (2,4) 


is  the  new 


minimal  violator  and  it  is 


eas- 


j 


L  (  2 , 4)  ; 

1.3 

1.8 

5*5 

~  ”  5”  5 

1.8 

(2.5) 

5.2 

1 

l  ; 

3 

r  -  -i  ■  -  t - f 

12  3  4 


i 


Thus  we  pool  (2,4)  with  (2,3)  to  obtain 


Now,  (3,3)  is  the  new  minimal  violator  with  g£(3  3)  as 
given  in  expression  (4.1).  Pooling  (3,3)  with  (3,2),  we  then 
have  (4,4)  as  the  minimal  (only)  violator.  Pooling  (4,4)  with 
(3,4)  we  have  as  the  final  solution 


g 


L(i, j) 


4 


1.8  4  6.5  6.5 


j 


3 

2 


1.8  4  5.1  6 

1.8  2.5  5.1  5.5 


1 
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